Systems, apparatus and methods for delivery of location-oriented information

ABSTRACT

An augmented reality guiding system for delivery of location-oriented information, including a location server, a database coupled to the location server, and at least one mobile locating device. The database is configured to receive, store, customize and send target information relating to target points of interest. Each mobile locating device has a geographical positioning module configured to determine location information for that locating device, an imaging module configured to capture at least one image of at least some of a field of vision for that locating device, a spatial orientation detection module configured to determine spatial information of that locating device, a display module configured to provide visual feedback, and at least one wireless communication module configured to communicate with the location server. Each mobile locating device is configured to be able to receive target information relating to at least one target points of interest from the location server, and based on the target information and the location information and spatial information for that locating device, display the target information relating to the target point of interest by visually augmenting the at least one captured image on the display module.

RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application Ser.No. 61/022,677 filed on Jan. 22, 2008 and entitled MOBILE SYSTEM FORDELIVERY OF LOCATION-OREINTED INFORMATION, the entire contents of whichare hereby incorporated by reference.

FIELD

The embodiments described herein generally relate to systems, apparatusand methods for locating and tracking of objects, and in particular todisplaying location information relating to mobile targets.

INTRODUCTION

Geographical positioning systems are now regularly employed asnavigational aids to assist travelers reach their destination. It is notunusual for a traveler to seek assistance from a geographicalpositioning system to receive guidance and directions to arrive at aparticular destination.

Typically, prior to starting a trip, a user of a GPS device will input adestination into the GPS device, which typically stores a copy of a mapof the user's current location and the destination location. The inputof the destination may be done by providing a street address, orindicating a point of interest on a local map. In some instances,multiple destinations may be indicated. However, if a destination is notstationary, the user normally has to continuously manually update thesystem with the destination's current geographical position to ensurethat the system is directing the user to the desired destination.

GPS systems are also capable of determining the geographical location ofthe user of the system. This permits the system to update the user'slocation on a 2D or 3D map such that the user may be regularly informedof his/her present location relative to the map. This also permits thesystem to plot a path that may be used to reach the destination(s)entered and display that path on the map.

However, the path is normally displayed on a map that represents thereal world. Typically the maps of the routes do not correspond to a reallive view that a user on one of the transportation routes mightexperience. Accordingly, if a user is unskilled or unfamiliar withreading or interpreting maps, the user may experience difficulties inusing the map with the path overlay.

Accordingly there is a need in the art for improved systems, apparatusand methods for delivery of location based information.

SUMMARY

According to one aspect of the invention, there is provided an augmentedreality guiding system for delivery of location-oriented information,comprising a location server, a database coupled to the location serverand configured to receive, store, customize and send target informationrelating to target points of interest, and at least one mobile locatingdevice, each mobile locating device having a geographical positioningmodule configured to determine location information for that locatingdevice, an imaging module configured to capture at least one image of atleast some of a field of vision for that locating device, a spatialorientation detection module configured to determine spatial informationof that locating device, a display module configured to provide visualfeedback, and at least one wireless communication module configured tocommunicate with the location server; wherein each mobile locatingdevice is configured to be able to receive target information relatingto at least one target points of interest from the location server, andbased on the target information and the location information and spatialinformation for that locating device, display the target informationrelating to the target point of interest by visually augmenting the atleast one captured image on the display module.

The target information may include at least one of location informationindicative of a location of each target point of interest, spatialorientation information indicative of spatial information relating toeach target point of interest, permission information for determiningwhether the target information is accessible or relevant, and groupinginformation about how each target point of interest is organized.

At least one of the target points of interest may be a mobile locatingdevice.

Each locating device may be further configured so that, when a targetpoint of interest is not within the field of vision the visuallyaugmenting of the at least one captured image includes indicating adirection from the locating device to a location of the target point ofinterest.

At least one of the target points of interest may be non-mobile.

Each locating device may configured to store target information relatingto at least one non-mobile target in the data storage module locally onthe locating device.

Each locating device may be configured to allow target points ofinterest to be added and stored in at least one of the data storagemodule and the location server.

In some embodiments, the imaging module, geographical positioningmodule, spatial orientation detection module, input module, displaymodule, and communication module are integrally coupled. In otherembodiments, the geographical positioning module and spatial orientationdetection module are removably coupled to the locating device.

In some embodiments, the geographical positioning module and spatialorientation module are configured to communicate with the input module,display module, and communication module using a wireless interface.

According to another aspect of the invention, there is provided a mobilelocating device for an augmented-reality guiding system, comprising ageographical positioning module configured to determine a locationinformation for the locating device, an imaging module configured tocapture at least one image of at least some of a field of vision for thelocating device, a spatial orientation detection module configured todetermine spatial information for the imaging module of the locatingdevice, a display module configured to provide visual feedback, and atleast one wireless communication module configured to communicate with alocation server, wherein the location server is in communication with adatabase configured to receive, store and send target informationrelating to target points of interest, the mobile locating device isconfigured to be able to receive target information relating to at leastone target points of interest from the location server; and based on thetarget information and the location information and spatial informationfor the locating device, display the target information relating to thetarget point of interest by visually augmenting the at least onecaptured image on the display module.

DRAWINGS

For a better understanding of the present invention and to show moreclearly how it may be carried into effect, reference will now be made,by way example, to the accompanying drawings, in which:

FIG. 1 is a block diagram of a locating device and a location serveraccording to one embodiment of the invention;

FIG. 2 is a perspective view of a locating device according to oneembodiment at an exemplary scene;

FIG. 3 is a diagram of a display module of the locating device of FIG. 2displaying a visual representation of the exemplary scene;

FIG. 4 is a perspective view of a locating device according to anotherembodiment at another exemplary scene;

FIG. 5 is a block diagram of a locating device, server and a positioningstation according to another embodiment;

FIG. 6 is a perspective view of a locating device according to anotherembodiment;

FIG. 7 is an exemplary view of a user input interface and a graphicaluser interface for a locating device;

FIG. 8 is a table showing a mapping of various functions to user inputbuttons in one embodiment;

FIG. 9 is an example of a visual overview of various display screens ona locating device in one embodiment;

FIG. 10 is a table listing some information that a locating device mightprocess according to one embodiment;

FIG. 11 is a block diagram displaying a tracking function according toone embodiment;

FIG. 12 is a block diagram of a system for delivery of location-orientedinformation according to one embodiment; and

FIG. 13 is an exemplary circuit diagram according to one embodiment.

DESCRIPTION OF VARIOUS EMBODIMENTS

Turning now to FIGS. 1 to 3, illustrated therein is a mobile system forthe delivery of location-oriented information having at least onelocating device in accordance with one embodiment of the invention.

Referring more specifically to FIG. 1, the system as shown includes afirst locating device 10, a wirelessly accessible server 50, and asecond locating device 60 similar to the first locating device 10. Insome other examples, the system may include more than two locatingdevices.

According to this embodiment, the locating device 10 includes a mobiletelecommunications device 20, a heading and spatial orientationdetection module 30, and a geographical position detection module 40.The mobile telecommunications device 20 includes a main program 21, animaging module such as a camera 22, a user input interface 23 such as akeypad, a data storage module such as a database 24, a display module25, a first wireless communication module 26 such as a wireless Internetinterface, and a second wireless communication module 27 such as aBluetooth interface.

In another embodiment, the imaging module may be a transparent portionof the locating device which may be shaped similar to a thin card.

The heading and spatial orientation detection module 30 may include aheading and spatial orientation sensor 31, such as a tilt-compensateddigital compass, and a wireless interface 32 such as a Bluetoothinterface.

The position detection module may include a position sensor 41, such asa global positioning system (GPS) receiver, and a wireless interface 42such as a Bluetooth interface.

During use, the wirelessly accessible server 50 handles sharing of thelocation data of the locating device 10 and locating device 60. Theserver 50 generally has a server program 51, a database 52 and awireless interface 53, such as a wireless Internet interface.

The server 50 is generally configured to receive, store and send targetinformation relating to various target points of interest. A targetpoint of interest may be one or more mobile locating devices, or anon-mobile target. For example, in FIG. 2, target points of interestinclude “Car” and “Home”.

The server 50 may also store the target information in a database 52. Insome embodiments, the target information may include locationinformation about the target points of interest (e.g. each locatingdevice or other mobile or non-mobile targets).

The location information generally includes information indicative ofthe geographical location of each target point of interest. In oneexample, the location information could be longitude and latitudeinformation. In other examples, the location information might beinformation relating to GPS indoor extension systems such aspseudolites, GPS reradiating systems, or other proximity detectionsystems such as ultrasound, infrared, RFID, WLAN, Bluetooth.

In addition to the location information, the target information may alsoinclude spatial orientation information. The spatial orientationinformation is generally indicative of the positioning of an imagingdevice (such as the camera 22) so that visual images captured by theimaging device may be associated with a particular orientation. Thevisual images captured by the imaging device generally represent atleast part of a user's field of vision when the image is captured.

The target information may also include organization informationincluding permission information related to target point of interest(e.g. whether the target point of interest wants to share its location,or share different versions of information depending on the queryinguser's properties, such as whether the user has a deluxe account or abasic account). The organization information may also include groupinginformation relating to a number of locating devices such as a “bestfriends” group, as will be described in greater detail below.

The target information may also include information regarding the targetpoint of interest such as an associated user's name. In otherembodiments, target information may include additional description orinformation associated with a particular target. For example, if thetarget is a store, target information might include hours of operation.In another example, if the target is a booth or a display at aconvention, the target information might include a vendor associatedwith that booth. In another example, target information may containadvertising or promotional information.

Target information may not necessary be in just text form. Targetinformation may contain multimedia content such as audio, video orphotos. For example, if a target is a display at a museum, the targetinformation may include audio commentary explaining the display. Inanother example, if the target is a movie theatre, target informationmay include video trailers of current movies that may be screening.

Now referring to FIG. 2, the locating device 10 is shown in one mode ofoperating in an example where the user tracks two objects or targetpoints of interest labeled “Car” 103 and “Home” 104. The locating device10 includes the heading and spatial orientation detection module 80 andthe position detection module 90, both affixed to the mobiletelecommunications device 70 by a harness 100.

The locating device 10 captures a video feed of an actual scene 101 witha camera 102 (including one or more images of the actual scene 101) anddisplays the video feed on a view screen 71 of the display module 25with location data overlaid thereon.

Note that in this example, the target point of interest “Home” is muchfurther away from the locating device (5 km) than the target point ofinterest “Car” (20 m). Accordingly, in some embodiments, the targetpoint of interest “Home” 104 may not be fully captured due to thelimited field of view of the camera 102, while the target point ofinterest “Car” 103 (which is generally much closer) may be fully ornearly-fully captured by the camera 102.

The content of the view screen 71 may be shown more clearly in FIG. 3.In some embodiments, the coordinates of the locations of “Home” 104 and“Car” 103 may be entered into the locating device 10 and stored in thedatabase 24. This may be done, for example, by directly specifyingcoordinates of each target point of interest or by detecting and savingthe coordinates for each target point of interest when the locatingdevice was at the location of “Home” or at “Car”.

Since the target point of interest “Home” may not be fully captured, itmay not be visible on view screen 110. Accordingly, its direction may beindicated by a 3D arrow 112 pointing to the actual direction of “Home”with respect to the view of camera 102 (as shown in FIG. 3). In someembodiments, the distance 114 from the locating device 1 to the targetpoint of interest “Home”, and its corresponding label 113, may also bedisplayed. The distance 114 and position of the 3D arrow 112 may becomputed by the main program 21 using heading and spatial orientationdata with respect to the view of the camera 102 using the heading andspatial orientation module 30, positional data of the locating device 10from position detection module 40 and the positional data of thetargeted objects stored in database 24.

However, since the target point of interest “Car” is generally withinthe view of the camera 102 and can be fully captured, it may bepresented with an indicator/mark 115 such as a circle drawn generallyover where the actual object (e.g. the Car) appears on the video feed onthe view screen 71.

Referring back to FIG. 1, in some embodiments the locating device 10 maybe used to track a mobile target object such as the second locatingdevice 60. For example, the position detection module 40 mayperiodically determines the position of the locating device 10 using theposition sensor 41. The main program 21 may periodically receive theposition data associated with the locating device 10 from the positiondetection module 40 (e.g. through the communication link provided by oneor more of the wireless interfaces 27 and 42). The main program 21 maythen store the position information in the database 24, and may alsosubmit the information to the server 50 via the wireless interface 26.The server program 51 may receive the position information through thewireless interface 53 and may subsequently store user positioninformation of locating device 10 in the database 52.

Similarly, the position of locating device 60 may also be determined,stored, and submitted to the server 50.

The heading and spatial orientation detection module 30 may periodicallydetermine the heading and spatial orientation of the locating device 10using the sensor 31, and the main program 21 may periodically receivethe heading and spatial orientation of the locating device 10 throughthe communication link over the wireless interfaces 27 and 32. The mainprogram 21 may then store the heading and spatial orientationinformation in the database 24.

To track the locating device 60, the main program 21 of the locatingdevice 10 may request the positional information of the locating device60 from the server program 51 (e.g. via the wireless interface 26). Theserver 50 may receive the request through the wireless interface 53.

If the locating device 10 has the required permissions to access thepositional information of the locating device 60 (e.g. if the locatingdevice 60 has authorized the locating device 10 to access its positionalinformation, or if the locating device 60 permits its positionalinformation to be publicly accessible to all locating devices), theserver program 51 may respond to the locating device 10 by sending thepositional information of the locating device 60 to the main program 21of the locating device 10 through the wireless interfaces 53.

However, if the locating device 10 does not have the requiredpermissions to access the positional information of the locating device60, the server program 51 may not send the positional information of thelocating device 60 and instead may notify the locating device 10 that itlacks the positional information of the locating device 60 and/or thatthe locating device 10 lacks suitable permissions to access thepositional information of the locating device 60.

Where the locating device 10 has the required permissions, uponreceiving the positional information of the locating device 60, the mainprogram 21 of the locating device 10 may store the received positionalinformation in the database 24. The main program 21 may then retrieve avideo feed from camera 22. Using the positional information of bothlocating devices 10 and 60 stored in the database 24, the main program21 may calculate the distance between the locating devices 10, 60.

Furthermore, using positional information of the locating device 10,heading and spatial information of the locating device 10, andpositional information of the locating device 60 stored in the database24, the main program 21 may calculate the direction to the locatingdevice 60 with respect to the field of view of the camera of thelocating device 10. With the calculated distance between the locatingdevices 10 and 60 and the direction to locating device 60 with respectto the camera of locating device 10, the main program 21 may determinewhether the locating device 60 is within the field of view of the cameraof the locating device 10.

The display of the positional and directional information about thelocating device 60 may be similarly displayed as the information aboutthe target points of interest (e.g. “Home” and “Car”) as previouslydescribed with respect to FIG. 3. If the locating device 60 is withinthe field of view of the camera of the locating device 10, the mainprogram 21 may mark or identify the locating device 60 on the video feeddisplayed on the view screen 71.

In some embodiments, the size of the indicator/mark may be related tothe actual distance between the locating devices 10, 60. For example,the indicator/mark may be small if the locating device 6 is far awayfrom the locating device 10, and may be large if the locating device 60is closer, and vice versa. In some embodiments, the size of theindicator/mark may change as the distance between the locating devices10, 60 changes.

In some embodiments, where the locating device 60 is not within thefield of view of the locating device 10, the main program 21 may displayan indicator (e.g. a two or three dimensional arrow) oriented in thedirection of the locating device 60 on the view screen 71.

According to some embodiments, the locating device 10 may track multipleother locating devices based on organizing information. Organizinginformation may include various criteria to determine if targetinformation relating to a target point of interest should be provided tothe mobile locating device 10. This may be helpful to locate a number ofmobile points of interests that meets a certain criteria. For example,organizing information may relate to the proximity in distance betweenthe locating device 10 and the target mobile points of interest, and thetarget information may only be displayed (in some embodiments) whereinthe proximity is less than a certain threshold (e.g. less than 10 km, 1km or 100 m).

Referring to FIG. 11, in some embodiments a user X using the locatingdevice 10 may wish to locate other users A, B and C within a thresholddistance d of the user X. At a first time T₁ the user X may receivetarget information relating to user B and user C indicating that theseusers B, C satisfy the tracking criteria of the locating device 10 (e.g.within the threshold distance d). However, at time T₂, since both user Xand user B may be mobile, the distance between user X and user B is nowgreater than the threshold distance d. Accordingly, user X may no longerreceive information about user B on the locating device 10. Howeversince user A is now in range (e.g. within the threshold distance d), theuser X may receive target information about the user A.

While FIG. 11 illustrates one embodiment relating to criteria forfinding other mobile points of interest, (namely specifying a thresholddistance as one criterion), other criteria may be employed. For example,another criterion may be grouping information relating to a number oflocating devices such as a “best friends” group. For example, a user mayonly wish to identify himself or herself as available only to a selectedgroup of other locating devices.

Another exemplary criterion may relate to tracking of a particularlocating device so that the particular locating device may be readilylocated and monitored.

Yet another criterion may be a security setting or availabilityindication of the target point of interest. For example, at times a userof a locating device may not wish to be disturbed (e.g. when they arestudying or want to be alone) and therefore the user may set thelocating device to an “invisible” mode such that other locating deviceswill not obtain location information about that user.

Referring again to the embodiment shown in FIGS. 1 to 3, target objectsof interest may be organized in multiple ways, not all of which aremutually exclusive. For example, one way of organizing objects ofinterest may be to categorize objects by groups. A basic organizationmethod may be to categorize objects of interest into generic orpredefined groups such as “people” and “places”. Objects of interestsmay also be further organized into user-defined groups, such as a“restaurants” group or a “best friends” group. A generic group maypossibly contain people, places and/or other entities.

Another way to organize objects of interests may be to categorize themstatically or dynamically. In some embodiments, the organization ofobjects may be changed by the user or may be automatically changed bythe main program 21. For example, the user may be allowed to modify alist of objects of interest in a group using the user input interface23. Alternatively, the main program 21 may request information about theclosest ten friends within a 1 km radius from the server 50 and mayupdate the database 24 with this information, hence dynamically updatingand grouping objects of interests by criteria such as proximity.

Yet another way to organize objects may involve categorizing objects ofinterest by their current availability. In particular, not all objectsof interest may have updated position information in the database 52.For example, the locating device 6 may have been logged off or shut downfor an extended period of time, and therefore no current positionalinformation may be available in database 52. As such, the user of thelocating device 10 may be able to distinguish between target objects ofinterest that are currently available and those that are not available.

Other suitable methods or techniques for organizing objects may be usedalone or in combination with the above-noted techniques.

In some embodiments, the target objects of interest may be defined bymany characteristics, such as a name, a description, a position, one ormore group memberships and properties relating to the type of indicatoror mark and/or arrow used to display locational information on the viewscreen (e.g. different colors, shapes, etc.). Some or all of thesecharacteristics may be stored in the locating device's database. Some orall of these characteristics may be stored in the server database 52.

In some embodiments, users may or may not require permission in order toadd an object as a target object of interest. For example, adding arestaurant as an object of interest may not require permission, whileadding the locating device of a friend as an object of interest mayrequire permission. In some embodiments, such permissions may be grantedby server 50, by the object of interest itself, and/or by a user.

Turning now specifically to the server shown in FIG. 1, in someembodiments the server program 51 may facilitate the sharing ofcoordinates between locating devices and may control the sharing ofcoordinates. However, the coordinates stored in the server database 52may not be limited to coordinates of locating devices. Coordinates ofstatic target objects of interest (e.g. restaurants and other fixedlocations) may be stored in the server database for use by variouslocating devices. In some embodiments, locating devices may log into(and optionally out of) the system through the server as well (e.g.using a login name and password, for example).

In some examples, the mobile telecommunications device 20 may be a cellphone with a camera 22, a Bluetooth interface 27, Internet access 26,and may run a Java platform for mobile telecommunications devicessupporting video overlay. The sharing of user location data may befacilitated by the server 50 being an HTTP server running server-sidescripting (e.g. server program 51) written in a suitable programminglanguage (e.g. Perl).

The positioning module 40 may be a GPS receiver 41 with Bluetoothinterface 42. The heading and spatial orientation module 30 may be atilt-compensated digital compass 31 that outputs heading, roll, andpitch data via a universal asynchronous receiver/transmitter (UART)interface to a Bluetooth module 32. One example of a circuit diagram ofa circuit that connects the tilt-compensated-compass chip to theBluetooth chip and powers both these chips is shown in FIG. 13.

In some embodiments, the main program may use Java Specification Request(JSR) 135 to control the cell phone's camera, JSR 82 to access theBluetooth interface, JSR 139 to access the Internet (e.g. to communicatewith the server 5), JSR 184 to create 3D graphics, JSR 135 or JSR 234 tooverlay graphics on the video feed, and JSR 118 to present the graphicaluser interface.

The embodiment depicted generally in FIGS. 1 to 3 may have manyalternative configurations. For example, in some embodiments, the label113 may be defined by user or given a generic name by the locatingdevice.

In some embodiments, one or more objects may be tracked simultaneously,and their locational data may be shown on the view screen as a pluralityof arrows 112, labels 113, distances 114 and marks 115.

In some embodiments, the shapes, colors, and/or textures of the marks115 and the arrows 112 may be different and may be defined by either theuser or the locating device 10.

In some embodiments, the arrows 112 may be 2-dimensional or3-dimensional.

In some embodiments, locational information such as arrows 112, labels113, distances 114 and/or marks 115 may be auditory, and may include theuse of automated voice systems and/or varying the pitch and frequency ofvarious beeping sounds or other alerts.

In some embodiments, the locational information may include tactilefeedback, such as providing varying strengths and patterns of vibrationto the locating device 10.

In some embodiments, locational information may one or more combinationsof visual, auditory, tactile and any other suitable notificationtechniques.

In some embodiments, communication links 26, 23, 32, 42 and 53 betweenvarious components may include any generally reliable communicationlinks (e.g. any wired communication links such as serial/parallelinterfaces, and wireless communication links such as the WiFi andBluetooth links).

In some embodiments, the communication links 26 and 27 may be combinedor duplicated to provide increased redundancy.

In some embodiments, the locating device 10 may be configured to includemore than one communication link to accommodate various communicationstandards (e.g. CDMA, TDMA, GSM, etc.).

In various embodiments, all of the components generally shown in FIG. 1may be integrated together or separated apart into various modules orgroupings. For example, the camera 22 may not be a part of the mobiletelecommunications device 20 (but instead may be provided as a separatemodule), the display 25 may be separated from the camera 22 and/or themobile telecommunications device 20, and the heading and spatialorientation detection module 30 may be integrated with the positiondetection module 40 as one component or module.

In some embodiments, the system as illustrated in FIG. 1 may have morethan one server 50 to provide for load balancing and redundancy.

In some embodiments, the system as illustrated in FIG. 1 may beconfigured to be used in outdoor environments, indoor environments (asgenerally described with reference to FIG. 4) or both.

In some embodiments, various types of one or more suitable positiondetection modules 40 (e.g. ultrasonic position detection modules and GPSreceivers) may be included in the locating device 10 for all/mostlyoutdoor usage, all/mostly indoor usage or both indoor and outdoor usage.

In some embodiments, the main program 21 may not provide distanceinformation 114 to the target object of interest, but may only providedirection information.

In some embodiments, the main program 21 may not determine straight-linedirectional information to the target object; instead, it may use staticpreloaded information (e.g. static maps) or dynamic information (e.g.live information about traffic video feeds from sources such as theInternet, and other accessible information) to determine and provide tothe user a restricted path, such as a path restricted to roads,pedestrian walkways, sidewalks, paved paths, or any other pathsgenerally free of obstacles and subject to user preferences.

In some embodiments, other devices capable of capturing and providing avideo feed on the view screen 71 may be used, including one or moreimages from satellite, or a juxtaposed video feed from individual videofeeds from one or more stationary cameras.

In some embodiments, each target object of interest may either be astationary object or may be a mobile object.

In some embodiments, the system may have internal live video feeds uponwhich additional information can be overlaid. Labels may be overlaid ontop of the corresponding target objects in the video feed on thedisplay. Directional information relating the location of the userrelative to the target object may be overlaid onto the video feed on thedisplay.

In some embodiments, the system may be configured to determine thedistance between the target object and the user within a maximum errorof 20 meters. In other examples, the maximum error may be 10 meters. Inyet other examples, the maximum error may be 45 meters. In yet otherexamples, the maximum error may be 90 meters.

In some embodiments, the system may have video capture and playbackcapabilities.

In some embodiments, the system may provide a search function forsearching for target objects of interest. The search function may beprovided through a graphical user interface displayed on the view screen71.

Referring now to FIGS. 5 and 6 generally, a mobile system for thedelivery of location-oriented information is shown in accordance to asecond embodiment of the invention. This embodiment is generallyconfigured for use in an indoor environment, such as for a museum tourguide application or within a trade show.

As shown, the system includes a locating device 130 having a mainprogram 131, a camera 132, a user input interface 133 such as a keypad,a database 134, a display module 135 (which may include peripherals suchas a view screen), a heading and spatial orientation detection sensor137 (e.g. a tilt-compensated digital compass), a position detectionsensor 138 (e.g. an ultrasonic positioning mobile transceiver), a firstcommunication link 139 (e.g. an ultrasonic mobile transducer), and asecond communication link 136 (e.g. a wireless Local Area Network (LAN)interface).

The locating device 130 may also be given a unique identification numbersuch as a serial number.

The system further includes a server 140 which may be wirelesslyaccessible for handling and sharing mobile locational data, and at leastone positioning station 150 for enabling positional detection. Theserver 140 generally has a server program 141, a database 142 and acommunication link 143 (e.g. a wireless LAN interface). Each positioningstation 150 generally includes a transceiver 151 (e.g. an ultrasonictransceiver) and a communication link 152 (e.g. an ultrasonictransducer).

As illustrated in FIG. 4, this system may be useful for indoor or mostlyindoor applications, such as a tour guide application in a museum. Usersmay each have one mobile locating device 120, shown with keypad 121,view screen 122 and camera 132.

The database 142 of the server 140 may include information for exhibitswithin the museum, as well as the locations of locating devices. Themain program 141 may include enhancements to facilitate the tour guideapplication, such as a keyword search 123 that may be used present alist of relevant exhibits (based on search keywords, etc.). Each exhibitmay have an associated description 126, distance information 125, anddirectional information 124, such as a 3D directional arrow.

Exhibits within the camera's field of view may have indicators/marks 127shown on the view screen 122 for highlighting the exhibits on the viewscreen. When a particular exhibit falls within a central zone 128 of thecamera's entire field of view, the locating device 130 may display ahighlighted indicator/mark 129 and specific descriptions 1210 associatedwith that exhibit.

In some embodiments, users desiring to track mobile objects (e.g. theircompanions or a tour guide) may use the corresponding locating device'sserial number to dynamically locate the mobile object.

When using ultrasonic positioning transceivers and transducers, theultrasonic positioning mobile transceiver 138 may send ultrasound burstsat various times (e.g. each time a position update is desired, forexample at a frequency of 1 Hz), and may listen for echoes from eachultrasonic positioning station transceiver 151 located at variouslocations within the premises.

In some embodiments, the sound bursts may be encoded with the locatingdevice's serial number. Each ultrasonic positioning station transceiver151 may listen for the ultrasonic bursts, and may check if the burstoriginated from a specific ultrasonic positioning mobile transceiver 138with the use of the encoded serial number.

If the burst came from an ultrasonic positioning mobile transceiver 138,the ultrasonic positioning station transceiver 151 may retransmit theburst, along with additional information such as its own station IDadded to the burst. This may prevent the echoed bursts from beingre-echoed by another station, since the other ultrasonic positioningstations may be programmed to ignore bursts containing a particularstation ID.

At least one ultrasonic positioning station transceiver 151 may bedistributed in each room so that, regardless of where the ultrasonicpositioning mobile transceiver 138 is located, at least four ultrasonicpositioning stations transceivers 151 are within listening range.Practically, the actual density of station transceivers 151 may begreater, since obstacles (e.g. the user's head/body or other persons inthe room) may block one side of the ultrasonic positioning mobiletransceiver 138.

After sending an initial burst, each mobile transceiver 138 may keep atime stamp of the time the burst was sent, and then listen forultrasonic bursts and only capture those that contain its own serialnumber and a station ID. Echoes from at least four station transceivers151 may give enough positional information for the locating device 130to compute its position with respect to these ultrasonic positioningstations using a position determining techniques (e.g. trilateration,triangulation and multilateration).

The locating device 130 may submit this information to the server 140,which may compute a coordinate, record this coordinate in its database142, and then send this information back to the locating device 130.

The embodiment depicted in FIGS. 5 to 6 may have many alternativeconfigurations, for example any one or more of the alternativeconfigurations generally described above.

Referring now to FIG. 6, a mobile system for the delivery oflocation-oriented information is shown in accordance to a thirdembodiment. The system may include a detachable separate module for usewith smart phone/PDA type mobile telecommunications devices.

As shown, the system includes of an accessory 160, a mobiletelecommunications device 170 and a server 180 similar to serversdepicted in FIG. 1 or 6, such as a wirelessly-accessible server tohandle optional sharing of mobile locational data.

The accessory 160 generally includes an electronic component 161 havinga position, heading, and spatial orientation module such as a part/allof a GPS receiver in combination with a part/all of a tilt-compensateddigital compass and a communication link such as a Bluetooth wirelessinterface for communicating with mobile telecommunications device 170.The system also includes a second electronic component 163 including apart/all of a GPS receiver in combination with a part/all of atilt-compensated digital compass and a communication link (e.g. a cable)for communicating to the first electronic component 161.

This system may be useful for users that already have a mobiletelecommunications device. In this case the user only needs to purchasethe extra hardware components 160 and the corresponding software may bedownloaded to the mobile telecommunications device provide the desiredset of user features.

The user's mobile telecommunications device 170 may have all thecapabilities described for mobile telecommunications device 20 inFIG. 1. Additionally, the mobile telecommunications device 170 may alsohave a communication link, such as an earphone jack or serial port (USB)for coupling with the accessory 160. In other examples, the earphonejack may be used as a hardware mechanism to attach the accessory 16 tothe mobile telecommunications device 170, and may not be a communicationlink.

The GPS receiver and tilt-compensated digital compass may be combinedinto one electronic component 161 and may output data to the mobiletelecommunications device 170 through a communication link, such as awireless Internet interface, and which may also be combined into theelectronic component 161.

Accordingly, the electronic component 161 may include a GPS receiver,processing components for a digital compass, a Bluetooth chip forBluetooth wireless communication, and/or a battery source. Theelectronic component 161 may be connected to a second electroniccomponent 163 by a connector 162 such as a cable.

The electronic component 163 may be smaller than the electroniccomponent 161 and may house the sensor chips of the digital compass. Thesmaller module may be further built into the base of a plug 164 that maybe compatible with the mobile telecommunications device 170, which mayalso provide a jack extension 165 for other conventional uses.

The embodiment depicted in FIG. 6 may have many alternativeconfigurations, for example one or more of the alternativeconfigurations described above.

Referring now to FIGS. 8 to 10, illustrated therein is an example of thedesign of the system's graphical user interface (GUI) and user inputinterface shown on a Motorola i880 mobile telecommunications device.

Since cell phones may have small screens and minimal computing power,the screens may be small, both in terms of display area but also interms of the number of available user input mechanisms.

FIG. 7 displays a portion of the Motorola i880 emulator. In thisembodiment, the phone may be limited to four basic function buttons: theLeft, Right, OK, and Menu buttons. A “Left” action may be executed whenthe Left Button is pressed. Similarly, the “Menu” and “Right” actionsmay be mapped to the Menu and Right Buttons, respectively. The OK Buttonmay be mapped to yet another action, if applicable to that specificscreen (e.g. a “Select” action).

If there are more desired commands, they may be accessible by pressingthe Menu button. This built-in Menu functionality may be automaticallyimplemented by the Motorola i880.

In another example, a SE W580i mobile telecommunications device may beused.

FIG. 8 generally lists an example of the mapping of functions to userinput buttons, for various main program classes.

FIG. 9 illustrates an example of a visual overview of the screens thatmay be involved. The arrows in FIG. 9 generally indicate an action thatthe user may take by pressing a button on the phone. The text directlyunderneath each screen may represent the corresponding Java class in thedesign. In some embodiments, the screens may be developed using MIDP 2.0(JSR 118).

In order to determine the response and performance of the 3D graphicsalgorithm, a test sequence may be developed to generate the relativedistance and angle information for situations encountered while usingthe system described the embodiments above. Referring generally to FIG.10, a summary of the input data format, size, and description of theinput test data is shown. These parameters may be used to determine aposition of the target on an captured image. The test cases may beseparated into three main categories.

One of the categories is when the target is very close to the object.This may be classified as when the object is within 0.02 m (which may bethe finest available precision of the GPS measurement) of the user inthe North, South, East, or West directions, and when the user's phonehas small heading, roll, and pitch angles (the finest precision may be0.1° depending on the device chosen).

Another category is when the target is at a reasonable distance from thelocating device. For example, reasonable distances may be when thetarget is located at 1 km North, from the locating device at varyingincrements from 5° up to 60° for each of the maximum value of heading,pitch, and roll, when the target is located at 1 km North, 1.5 km Eastat varying increments of 5° up to 60° for each of value of heading,pitch, and roll, and when the target is located at 1 km North, 1.5 kmEast, 200 m Above at varying increments of 5° up to 60° for each valueof heading, pitch, and roll.

Another category is when the target is very far from object. This may beclassified as when the target is on the other side of the earth or whenthe curvature of the earth would interfere with locating the object.This may be used to determine if the algorithm generates meaningful datafor locations that may be very far away. The circumference of the eartharound the equator approximately 40076 km; thus half of that valueshould generally be the farthest valid relative distance from theuser-unit to any object on the earth.

In some examples, there is provided a mobile telecommunications deviceapplication that renders location-related information pertaining totarget points of interest through an augmented reality interface. Morespecifically, the target points of interest may include, but are notlimited to, people (e.g., friends, family, and pets) and places (e.g.,grocery stores and parking spots). The positions of these target pointsof interest may also be in many forms, e.g., street addresses, as wellas longitudinal and latitudinal coordinates.

In some embodiments, it is envisioned that a location server may provideaccount management and information services to users of locatingdevices. For example, these features might include ability to keep alist of contacts, and share location data with contacts to track eachother. Additionally, the server might facilitate integration of thirdparty sites such as web-based social networking tools (e.g. Facebook) toprovide at least some organizing information.

In some embodiments, the system may be used to provide guided tours ofamusement parks, parks, zoos, historic sites, museums, or other pointsof interest. In other embodiments, the locating device might be used toguide passengers to their destinations within transportation hubs suchas train stations, bus terminals and airports. In other embodiments, thelocating device might be used to track fellow travel companions whileinside a park or a part of an escorted tour group.

While the above description includes a number of exemplary embodiments,many modifications, substitutions, changes and equivalents will nowoccur to those of ordinary skill in the art.

1. An augmented reality guiding system for delivery of location-orientedinformation, comprising: a) a location server; b) a database coupled tothe location server and configured to receive, store, customize and sendtarget information relating to target points of interest; and c) atleast one mobile locating device, each mobile locating device having ageographical positioning module configured to determine locationinformation for that locating device, an imaging module configured tocapture at least one image of at least some of a field of vision forthat locating device, a spatial orientation detection module configuredto determine spatial information of that locating device, a displaymodule configured to provide visual feedback, and at least one wirelesscommunication module configured to communicate with the location server;d) wherein each mobile locating device is configured to be able toreceive target information relating to at least one target points ofinterest from the location server, and based on the target informationand the location information and spatial information for that locatingdevice, display the target information relating to the target point ofinterest by visually augmenting the at least one captured image on thedisplay module.
 2. The system of claim 1, wherein the target informationincludes at least one of location information indicative of a locationof each target point of interest, spatial orientation informationindicative of spatial information relating to each target point ofinterest, permission information for determining whether the targetinformation is accessible or relevant, and grouping information abouthow each target point of interest is organized.
 3. The system of claim1, wherein at least one of the target points of interest is a mobilelocating device.
 4. The system of claim 1, wherein each locating deviceis further configured so that, when a target point of interest is notwithin the field of vision the visually augmenting of the at least onecaptured image includes indicating a direction from the locating deviceto a location of the target point of interest.
 5. The system of claim 1,wherein at least one of the target points of interest is non-mobile. 6.The system of claim 1, wherein each locating device is configured tostore target information relating to at least one non-mobile target inthe data storage module locally on the locating device.
 7. The system ofclaim 1 wherein each locating device is configured to allow targetpoints of interest to be added and stored in at least one of the datastorage module and the location server.
 8. The system of claim 1 whereinthe imaging module, geographical positioning module, spatial orientationdetection module, input module, display module, and communication moduleare integrally coupled.
 9. The system of claim 1, wherein thegeographical positioning module and spatial orientation detection moduleare removably coupled to the locating device.
 10. The system of claim 9wherein the geographical positioning module and spatial orientationmodule are configured to communicate with the input module, displaymodule, and communication module using a wireless interface.
 11. Amobile locating device for an augmented-reality guiding system,comprising: a) a geographical positioning module configured to determinea location information for the locating device; b) an imaging moduleconfigured to capture at least one image of at least some of a field ofvision for the locating device; c) a spatial orientation detectionmodule configured to determine spatial information for the imagingmodule of the locating device; d) a display module configured to providevisual feedback; and e) at least one wireless communication moduleconfigured to communicate with a location server; f) wherein thelocation server is in communication with a database configured toreceive, store, customize and send target information relating to targetpoints of interest, the mobile locating device is configured to be ableto receive target information relating to at least one target points ofinterest from the location server; and based on the target informationand the location information and spatial information for the locatingdevice, display the target information relating to the target point ofinterest by visually augmenting the at least one captured image on thedisplay module.
 12. The mobile locating device of claim 11, wherein thetarget information includes at least one of location informationindicative of a location of each target point of interest, spatialorientation information indicative of spatial information relating toeach target point of interest, permission information for determiningwhether the target information is accessible or relevant and groupinginformation about how each target point of interest is organized. 13.The mobile locating device of claim 11, wherein at least one of thetarget points of interest is another mobile locating device.
 14. Themobile locating device of claim 11, wherein each locating device isfurther configured so that, when a target point of interest is notwithin the field of vision the visually augmenting of the at least onecaptured image includes indicating a direction from the locating deviceto a location of the target point of interest.
 15. The mobile locatingdevice of claim 11, wherein at least one of the target points ofinterest is non-mobile.
 16. The mobile locating device of claim 11,wherein the locating device is configured to store target informationrelating to at least one non-mobile target in the data storage modulelocally on the locating device.
 17. The mobile locating device of claim11, wherein the locating device is configured to allow target points ofinterest to be added and stored in at least one of the data storagemodule and the location server.
 18. The mobile locating device of claim11, wherein the imaging module, geographical positioning module, spatialorientation detection module, input module, display module, andcommunication module are integrally coupled.
 19. The mobile locatingdevice of claim 11, wherein the geographical positioning module andspatial orientation detection module are removably coupled to thelocating device.
 20. The mobile locating device of claim 19 wherein thegeographical positioning module and spatial orientation module areconfigured to communicate with the input module, display module, andcommunication module using a wireless interface.